This programming reading is provided as part of week three of “Clinical Natural Language Processing”, the fourth course in the Coursera Clinical Data Science Specialization created by the University of Colorado Anschutz Medical Campus and supported by our industry partner Google Cloud.

Last week you learned how to create regular expressions and work with text in R and the tidyverse. This week we are focused on how to apply the “Note Section Technique” for text processing. This approach uses the context provided by note section to understand the meaning of text matches. For example, the word “penicillin” in the Medication section means the patient is likely taking the drug, whereas the same text in the Allergy section, means that patient is likely not taking the drug as they are allergic and may have had a negative reaction to it in the past.

In these examples, we will use clinical notes created by medical transcriptionists that are freely available from www.MTSamples.com. Although these notes were created as training examples rather than notes for real patients, I can confidently say that these notes are very realistic to what you will see as a clinical data scientist. In some cases we have edited the text slightly to provide you with 1) additional messiness common in provider created notes and 2) specific examples of negation and other false positives to demonstrate real-life text processing challenges. There are three data sets available on Google BigQuery for this course, all of which are stored in course4_data:

Getting Started

First, let’s set up our environment. We need three packages:

We also set up our connection to the Google BigQuery project to be able to access the data used in this course. If you aren’t familiar with these packages I highly recommend working through the R programming section of “Introduction to Clinical Data Science”, the first course in the Coursera Clinical Data Science Specialization.

library(tidyverse)
library(magrittr)
library(bigrquery)

con <- DBI::dbConnect(drv = bigquery(),
                      project = "learnclinicaldatascience")
radiology_reports <- tbl(con, "course4_data.radiology_reports") %>% 
  collect()
discharge_summaries <- tbl(con, "course4_data.discharge_summaries") %>% 
  collect()

Programming Examples

In this week of the course you are going to work through two text processing tasks. The first we will work together and the second you will work through as part of the the “Try it out for yourself” examples.

To apply the note section technique you will follow three steps:

  1. Identify the note type and note section of interest
  2. Extract the note section of interest
  3. Search the note section of interest

We will go through each of these steps in the following tabs.

Identify Note Types and Sections of Interest

The first step of the note section approach is to identify:

  1. The note that would have your topic of interest.
  2. The note section in that note type that would have your topic of interest.

For this example we want to find patients who have any type of cyst in their abdomen.

Note Type Selection

Cysts are typically found during medical imaging studies like CT scans and MRIs. The results of theses scans are found in imaging or radiology reports. There are a lot of radiology reports for different types of scans (e.g., CT, MRI, X-Ray, Ultrasound etc.) and different areas of the body (e.g., brain, abdomen, wrist, etc.).

Given our goal of finding cysts in the abdomen we would want to search for radiology reports of CT/MRI scans of the abdomen and pelvis. In this particular case we have 11 radiology reports for CT scans of the abdomen and pelvis only. In the real world you would want to determine if you want to also include MRI reports or if you really should focus on a single type of imaging study. For example, for your analytic question one type of scan may be more accurate or sensitive for your goal.

Note Section Selection

Within radiology reports there are a number of different common sections:

  • Type of Exam
  • Clinical History/Reason for Exam
  • Comparison Studies
  • Technique
  • Findings
  • Impression

The information about what was actually seen on the scan can be found in the Findings and Impression sections. Findings lists all observations made by the radiologist from the imaging study, while the Impression section focuses on the relevant findings given the reason for the exam.

Given that cysts are usually found incidentally and typically are not relevant to the reason for the exam, let’s focus our search on the Findings section.

Try it out for yourself:
  • What type of notes would you expect to record a patient’s history of hypertension?
  • What sections in those notes would you expect to find a patient’s history of hypertension?
  • What note sections would you avoid using? Why?

Please note that we will reveal some of the answers to these questions in the future sections, so try answering these now!

Return to Top

Extract Note Section of Interest

There are two main methods for extracting the note section of interest:

  1. Delimiter Method
  2. Header Method

Delimiter Method

This method relies on each section being separated by some standard delimiter - for example each new section starting after two new line characters, something like:

CT Abdomen and Pelvis

Reason for Exam - Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.

Comparison - None

Or perhaps each section starts on a new line with a colon after the title like:

Exam Type: CT Abdomen and Pelvis
Reason for Exam: Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.
Comparison: None

What ever the particular formatting the goal is to figure out what the consistent feature is that defines each note section and to use that feature to split the note into those sections. In particular this approach follows two steps:

  1. Identify delimiter and split note into sections
  2. Extract section label for each section

1. Identify delimiter and split note into sections

Let’s take a look at our radiology reports to see if there are any obvious separators.

It looks like each section is separated from one another by white space. The tool that displays these pretty tables actually does some processing of the data to show spaces, so let’s take a closer look at a single note to see what the actual separator is:

Indeed it looks like each section is separated by two newline (\n) characters. We can use some cool functions from the tidyr and stringr packages to separate each note section.

radiology_reports %>% 
  separate_rows(TEXT, sep = "\n\n")

Code Explanation:The separate_rows() function in tidyr separates the text based on the pattern of two newline characters \n\n and gives each group of text it’s own row and labeled with the appropriate NOTE_ID.

2. Extract section label for each section

Now that we have the sections split into their own line, we want to pull out the section header for each row. In this case we can see that each header is split from the text of the section using a colon (:).

radiology_reports %>% 
  separate_rows(TEXT, sep = "\n\n") %>% 
  separate(col = TEXT, into = c("SECTION_TYPE", "SECTION_TEXT"), sep = ":", remove = TRUE, extra = "merge")

Code Explanation: In this code separate() is a function from tidyr that takes one column, separates by the text in sep= and puts those pieces into the number of columns specified by into=. The remove = TRUE argument gets rid of the original column. The extra = "merge" command means that if there were multiple separators in the note (let’s say the radiologist uses a colon somewhere else in the section), it would only split based on the first colon and the rest of section text would be kept together in a single block. 

## Warning: Expected 2 pieces. Missing pieces filled with `NA` in 3 rows [36, 41,
## 50].

Looks good, though we got a warning about three rows - let’s take a look:

radiology_reports %>% 
  separate_rows(TEXT, sep = "\n\n") %>%  
  separate(col = TEXT, into = c("SECTION_TYPE", "SECTION_TEXT"), sep = ":", remove = TRUE, extra = "merge") %>% 
  slice(c(36, 41, 50))

Code Explanation: The slice() function from dplyr allows you to select rows based on their position in the data frame. So we can look at the exact three rows that were identified in the error message: [36, 41, 50]. 

## Warning: Expected 2 pieces. Missing pieces filled with `NA` in 3 rows [36, 41,
## 50].

It looks like in these cases the header “EXAM:” was omitted resulting in the section text being listed as the exam type. In this case we can reformat these three rows and make them match the convention followed by the majority of other notes.

radiology_reports %>% 
  separate_rows(TEXT, sep = "\n\n") %>% 
  separate(col = TEXT, into = c("SECTION_TYPE", "SECTION_TEXT"), sep = ":", remove = TRUE, extra = "merge", 
           fill = "left") %>% 
  mutate(SECTION_TYPE = case_when(is.na(SECTION_TYPE) ~ "EXAM",
                                  TRUE ~ SECTION_TYPE)) %>% 
  slice(c(36, 41, 50))

Code Explanation: If we add the fill = "left" argument to the separate() command it will put text without a header into the SECTION_TEXT column and make the SECTION_TYPE NA. We can then use mutate() and case_when() to turn all NAs to “EXAM”, and leave the other values for SECTION_TYPE the same. Be very careful if you ever do this in the real world! You must make sure that all of the NAs really do have the same section type!

At this point then we can filter down to a list of the findings sections to pass on to the next step of the process - searching the note section of interest!

findings_delimitermethod <- radiology_reports %>% 
  separate_rows(TEXT, sep = "\n\n") %>% 
  separate(col = TEXT, into = c("SECTION_TYPE", "SECTION_TEXT"), sep = ":", remove = TRUE, extra = "merge", 
           fill = "left") %>% 
  mutate(SECTION_TYPE = case_when(is.na(SECTION_TYPE) ~ "EXAM",
                                  TRUE ~ SECTION_TYPE)) %>% 
  filter(SECTION_TYPE == "FINDINGS")
findings_delimitermethod

Try it out for yourself:

  • Is there a standard delimiter between sections in the discharge summary corpus? If so, what is the delimiter?
  • Split the notes based on that delimiter so that each note section gets its own row. How many rows does your data frame now have?
  • What delimiters can you use to separate the section header from the text in each section?
  • Does the section header delimiter work for every section? If not, what is happening with those sections?
  • What section headers are there that describe some form of clinical history about the patient?

Return to Top

Header Method

Although the section delimiter method is very convenient, the vast majority of clinical notes you encounter as a clinical data scientist will not be as consistently structured as these example notes. An alternative approach is to use the section header names in your notes and use those headers to process out the section you want to extract. This method has 3 steps:

  1. Find the section heading of interest
  2. Extract text after section heading of interest
  3. Identify section heading of next section and remove all content after that section.

Let’s walk through an example:

If your original note looks like:

Exam Type: CT Abdomen and Pelvis
Reason for Exam: Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.
Comparison: None
Technique: CT of the abdomen and pelvis performed without and with approximately 54 ml Isovue 300 contrast enhancement.

Let’s say your target is the reason for the exam. You would identify the section headers that match that area - for example: “Reason for Exam”, “History”, “Clinical Indication”, etc. Then you would extract all the text after that heading of interest and your new text snippet would be:

Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.
Comparison: None
Technique: CT of the abdomen and pelvis performed without and with approximately 54 ml Isovue 300 contrast enhancement.

This starts the text snippet with the section of interest, but still has all of the remaining sections in the note at the end of the snippet. Next we’d identify the titles of all sections after our section of interest. In this note it is “Comparison:”, in other notes perhaps is may be “Technique:” or “Findings:”. When you have identified all following sections headers, simply remove the text after that section header leaving you with just the section text we are interested in:

Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.

Now that you understand the general process let’s see how to actually perform these steps programatically.

1. Find the section heading of interest

We want to find the “Findings:” section of the radiology reports. First let’s just see how many notes have that exact text.

radiology_reports %>% 
  mutate(findings = str_detect(TEXT, regex("findings:", ignore_case = TRUE))) %>% 
  count(findings)

While the majority do have a section header called “Findings:”, 4 do not. Let’s look at these notes to see what heading they might have instead:

radiology_reports %>% 
  filter(!str_detect(TEXT, regex("findings:", ignore_case = TRUE)))

Code Explanation: Note that the exclamation point ! means not so the cases where you do not detect the term “Findings:” in the TEXT field.

It looks like these notes separated the findings based on location (abdomen vs pelvis) and labeled as “CT Abdomen:” and “CT PELVIS:” In this case, let’s plan on extracting both sections.

So across the different notes we will extract the following section headings:

  • Findings:
  • CT Abdomen:
  • CT Pelvis:

2. Extract text after section heading of interest

Now that we know which sections we want, let’s start by extracting all text after the section header of interest. To do this we will write a regular expression that matches the header name and then any text after the header using .*.

Because we have multiple headers I am going to use case_when() to extract these sections only for notes that have the header of interest. While I could write a regular expression that matches either header I find that level of complexity difficult to follow when I return to analyses months or years in the future. Feel free to use what makes the most sense to you.

As I showed in the regular expression programming example last week, when I do these kinds of manipulations I like to see the original content and our new section text just to make sure that I am getting exactly what I wanted from the manipulation.

radiology_reports %>% 
  mutate(SECTION_TEMP = case_when(str_detect(TEXT, regex("findings:", ignore_case = TRUE)) ~ 
                                    str_extract(TEXT, regex("findings:.*", ignore_case = TRUE, dotall = TRUE)),
                                  str_detect(TEXT, regex("ct abdomen:", ignore_case = TRUE)) ~ 
                                    str_extract(TEXT, regex("ct abdomen:.*", ignore_case = TRUE, dotall = TRUE)),
                                  TRUE ~ ""))

Code Explanation: I use case_when() to first detect if the note text has the word “findings:”. using the regex() function to allow a case-insensitive search with the ignore_case = TRUE argument. When I then go to pull out the text after the header using str_extract(), I change my regular expression to include .* after the header name. This should capture all content to the right of that header. However once quirk of the package is that in some regular expression engines the wildcard (.) does not actually match new line characters like or You can add the dotall = TRUE argument to regex() to make sure that the wildcard really does capture every character present.

Each of the SECTION_TEMP text begins with the section header of interest, which is exactly what we want. Let’s make this a permanent column in the data frame and move on to the next step!

radiology_reports %<>% 
  mutate(SECTION_TEMP = case_when(str_detect(TEXT, regex("findings:", ignore_case = TRUE)) ~ 
                                    str_extract(TEXT, regex("findings:.*", ignore_case = TRUE, dotall = TRUE)),
                                  str_detect(TEXT, regex("ct abdomen:", ignore_case = TRUE)) ~ 
                                    str_extract(TEXT, regex("ct abdomen:.*", ignore_case = TRUE, dotall = TRUE)),
                                  TRUE ~ ""))

Code Explanation: Reminder, the %<>% command from magrittr takes the data frame contents, processes it, and puts the results back into the original data frame. It is equivalent to: radiology_reports <- radiology_reports %>% mutate(...)

3. Identify section heading of next section and remove all content after that section.

The next step of the process is to quickly review a couple of notes to see what the section header is after our section of interest. If we look at the previous table, the next section after CT PELVIS in note 1 is “IMPRESSION:”, same for notes 2 and 3.

Our goal in this section is to remove any text after the next section header from the section. Let’s try it will impression and see if we captured each note.

radiology_reports %>% 
  select(NOTE_ID, SECTION_TEMP) %>% 
  mutate(SECTION_TEXT = str_replace_all(SECTION_TEMP, 
                                        pattern = regex("impression:.*", ignore_case = TRUE, dotall = TRUE), 
                                        replacement = ""))

Code Explanation: Just like we did in the second step we can add the .* argument to the end of our regular expression to capture all content after the next header. Again we can use dotall = TRUE to capture content on new lines as well.

Now we want to look at the results and see if there are any notes that have a different section after “Findings:” than “Impression:”. After reviewing all 11 notes it looks like these notes were really consistently formatted.

Let’s save these results.

findings_headermethod <- radiology_reports %>% 
  select(NOTE_ID, SECTION_TEMP) %>% 
  mutate(SECTION_TEXT = str_replace_all(SECTION_TEMP, 
                                        pattern = regex("impression:.*", ignore_case = TRUE, dotall = TRUE), 
                                        replacement = "")) %>% 
  select(NOTE_ID, SECTION_TEXT)


Tips and Techniques

In the real world you will almost certainly not be so lucky to only have a single header type as the next section header. You also will likely have hundreds to thousands of notes to process. You can’t count on being able to manually inspect every single note for these errors. Here a couple of tricks to help you inspect your data to catch outliers that haven’t been fully processed:

1. Sort by Section Length. You can imagine that if you failed to remove all text after your section of interest, the length of the text in the column is likely to be longer than those that have been processed. You can use this code to add a new column that has the section length (in character) that you use to sort by:

radiology_reports %>% 
  select(NOTE_ID, SECTION_TEMP) %>% 
  mutate(SECTION = str_replace_all(SECTION_TEMP, 
                                   pattern = regex("impression:.*", ignore_case = TRUE, dotall = TRUE), 
                                   replacement = "")),
         SECTION_LENGTH = str_length(SECTION)) %>% 
  arrange(desc(SECTION_LENGTH))

Code Explanation: To make it easier to follow this example I first used select() to only examine the NOTE_ID and SECTION_TEMP. In addition to the SECTION column I also added a column for SECTION_LENGTH that uses the str_length() function from stringr to get a character count of SECTION and then sort by that section using the arrange() function.

2. Look for Delimiter Type Text: Another method you can use is to filter to rows that have text that indicates section headers like a colon (:) or dash (-). Depending on the note type you could also look to see if there are additional newline characters that could be headers. Code that would do these filters is:

radiology_reports %>% 
  select(NOTE_ID, SECTION_TEMP) %>% 
  mutate(SECTION = str_replace_all(SECTION_TEMP, 
                                   pattern = regex("impression:.*", ignore_case = TRUE, dotall = TRUE), 
                                   replacement = "")) %>% 
  filter(str_count(SECTION, pattern = "[:\\-\n\r]") > 1)

Code Explanation: The str_count() function from stringr returns the number of matches found for a particular pattern in the column. We can then use a filter() function to select only those where the result of str_count() is greater than 1 (> 1).

As we can see we did pick up the notes that had the extra “CT PELVIS:” section (which we actually want for this analysis), as well as some off target matches for the dash for things like “12-mm” and “1-cm”.

3. First Extract the Next Section: A third approach is to first extract the next section from the column. For example:

radiology_reports %>% 
  select(NOTE_ID, SECTION_TEMP) %>% 
  mutate(NEXT_SECTION = str_extract(SECTION_TEMP, pattern = regex("impression:.*", ignore_case = TRUE, dotall = TRUE)))

This is useful if you have a lot of unique section headers as if there is an NA you know that you haven’t identified a section header for that note yet. However this is a bit of a dangerous approach for cases where section headers may be in the note, but just in a different order. For example if you had the following notes:

Note 1:

Exam Type: CT Abdomen and Pelvis
Reason for Exam: Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.
Technique: CT of the abdomen and pelvis performed without and with approximately 54 ml Isovue 300 contrast enhancement.

Note 2:

Exam Type: CT Abdomen and Pelvis
Reason for Exam: Nausea, vomiting, diarrhea for one day.
Fever. Right upper quadrant pain for one day.
Comparison: None
Technique: CT of the abdomen and pelvis performed without and with approximately 54 ml Isovue 300 contrast enhancement.

They both will extract text for Technique:.*, however in Note 2 Technique is not the very next section. You will want to pair this method with one of the other to make sure that you have actually captured the next section for all notes!

Try it out for yourself:

  • Write one or more regular expressions (as needed) to capture the section headers that describe some form of clinical history (i.e., that have the word “history” in their name) about the patient.
  • Try extracting the text after those section headers. Do all notes have one of those section headers?
  • If there is a note missing those section headers are there any other sections in those notes that might have the same information, but fail to use the word “history”? If so update your regular expression to include that section header.
  • What are the section headers for those sections immediately after the section of interest?
  • Use these section headers to extract just the sections of interest for further analysis in step 3.

Return to Top

Other Thoughts

At this point you may have noticed that the results from these two methods as we applied them are different. The Delimiter Method only returned findings sections for seven notes, whereas with the Header Method we identified that four notes had the findings labelled as “CT ABDOMEN:” and/or “CT PELVIS:”.

findings_delimitermethod %>% 
  nrow()
## [1] 7
findings_headermethod %>% 
  nrow()
## [1] 11

This was intentional as it highlights a real-world challenge when applying these techniques. As we have discovered, the Delimeter Method does not initially target the notes section of interest (only a delimiter). Because of this, it may not be apparent that anything is missing. Conversely, we have to search for terms of interest when utilizing the Header Method, which makes it more likely that we will find these discrepancies as we work through the process. Ultimately, when using the Delimiter Method, additional data quality checks (like checking row count) may be necessary to make sure that you get the section of interest for all notes.

When using either method, the overall lesson I hope you remember is that it’s important to really look at your data and ask yourself if you have really have captured everything you need. Systematically applying data quality checks along your data wrangling tasks is critical to catch edge cases. That said, it’s also important to realize that with large, real-world data sets your goal may not be to get 100% accuracy. You’ll find that about 80% of your data can be handled programmatically fairly easily, but that last 20% is almost entirely unique or semi-unique situations that could takes months to properly handle. Knowing when to stop is as important as knowing when to keep looking for exceptions. Knowing where this line is will be a skill you develop as you work with data that you may adjust for different project needs/goals.

Return to Top

Search Note Section of Interest

As mentioned in the “Other Thoughts” section in “Extract Note Section of Interest” the Header Method provided the most complete data, so we will use those results for this final step of the process.

Now that we have the note sections extracted we can search for our term of interest within that note section. We want to find patients who have a cyst on their CT scan. Last week we developed regular expressions for terms indicating cysts: (?<![a-zA-Z])cyst(s|ic)?(?![a-zA-z]) (See Regular Expressions and Text Processing in R for more!)

We can search the note sections of interest and create a variable that tells us if a cyst was mentioned or not.

findings_headermethod %>% 
  mutate(CYST = case_when(str_detect(SECTION_TEXT, regex("(?<![a-zA-Z])cyst(s|ic)?(?![a-zA-z])", ignore_case = TRUE)) ~ 1,
                          TRUE ~ 0))

Try it out for yourself:

  • How many patients had a diagnosis of hypertension listed in one of the sections with some form of clinical history about the patient?

Return to Top